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Toward Establishing Relationships Betvaen Essential 
and Higher order Teaching Skills 



Abstract 



Nineteen teachers participated in a single-group pretest-posttest 
design to explore the relationship between essential and higher 
order teaching skills. The teachers were observed on both 
occasions using an instrument designed to measure essential 
teaching skills (FPMS) and an instrument designed to measure 
teaching of higher order thinking skills (THOT) . Teachers' 
participation in a training program in the teaching of higher 
order thinking resultea in substantial increases in the THOT 
scores, but no increases in the FPMS scores. Additionally, non- 
significant correlations were obtained between the two scores on 
both occasions. The results were interpreted in terms of 
implications for teacher education and teacher evaluation. 



ERIC 



3 



Empirical research on teaching has shown that specific 
teacher performance variables such as use of time^ handling of 
materials, questioning, and maintaining discipline, are related to 
improvement in student achievement and conduct. In addition, 
students* acquisition of higher-order thinking skills has been 
related to specific teacher performance that leads students to (a) 
identify subject-related problems and variables, (b) hypothesize 
and test hypotheses, and (c) validate generalizations - 

Improvements in student achievement and conduct, and student 
acquisition of higher-order thinking skills are high priorities of 
education. This claim is supported by the emphasis given to 
basic academic instruction in the early 1980 *s, an emphasis 
resulting in large part from the finding in A Nation at Risk 
(198 3) that standardized achievement test scores were dropping. 
Further support is given by the recent emphasis on developing 
skills in reasoning, thinking critically, and problem-'solving 
suggested by Educating Americans for the 21st Century (1984). 

In their pursuit of improvements in student achievement, 
conduct, and higher-order thinkinr% educators may seek 
improvements in facilities, equipment, curriculum, or teacher 
performance. Although all of these factors may affect the desired 
improvements, the performance of teachers is probably the most 
important. Empirical research (Evertson, 1980; McDonald, 1976; 
Soar, 1968; Stallings, 1981) supports the primacy of teacher 
performance as a predictor of student learning. The empirical 
research on teacher effectiveness provides a solid basis on which 
to define effective teaching, examine current practice, attempt to 
improve practice, and evaluate the results of such attempted 
improvements. 

The purposes of this study were (a) to examine the 
relationship between teacher performance that is related to 
student achievement and conduct, and teacher performance that is 
related to student acquisition of higher order thinking skills; 
and (b) to examine the effects of a training program in teaching 
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higher order thinking skills on both sets of teacher performance 
variables. Previous research (Peterson, Kromrey, Borg, & Lewis, 
1990) suggested that (a) these two classes of teacher performance 
variables are relatively independent; (b) incidences of the 
behaviors related to the teaching of higher order thinking skills 
are relatively rare occurrences; and (c) the frequency of 
occurrence of teacher behaviors related to higher order thinking 
skills can be substantially increased through specific training. 
These conclusions were tentatively advanced by Peterson et al 
(1990), based upon a small sample of participating teachers. An 
additional purpose of this research, therefore, was to verify the 
findings of the Peterson et al (1990) study, using a larger sample 
of teachers. 

Knowledge Base 

Teacher performance that relates to student achievement and 
conduct, we have called essential teaching skills. These skills 
are considered essential because without discipline in the 
classroom, organization of instruction, and effective interaction 
with students, the literature indi'cates that student acquisition 
of knowledge is impaired. 

The domains, or areas, of essential teaching skills, 
delineated in Figure 1, were derived from process/product studies 
in which student achievement and conduct were the dependent 
variables. The results of the search of over 300 studies were 
organized into six domains, which include 34 concepts and 124 
indicators of teacher classroom performance (Domains: Knowledge 
Base of the Florida Performance Measurement System (FPMS) , 1983). 
Examples of specific observable teacher behaviors (indicators) for 
each domain are presented in Figure 1. 

Through a similar process, the domains, concepts, and 
indicators for teaching higher order thinking were derived from 
some 500 quantitative studies in which the dependent variables 
were student acquisition of procedures for reasoning, solving 
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problems, or critical thinking. Where the essential teaching 
skills help students to acquire knowledge of a subject and to 
recall and reproduce this knowledge, teaching for higher order 
thinking helps students to manipulate information stored in memory 
— to interrelate or rearrange the information to achieve a 
purpose, to solve a problem, to resolve an argument, or to make a 
prediction. The results of this literature search were organized 
into five domains, that include 15 concepts and 64 indicators. The 
five domains of Teaching for Higher Order Thinking (THOT) are 
Developing and Maintaining Flexibility and Student Awareness; 
Generating and Validating Generalizations; Assessing Arguments; 
Negotiating Issues and Solving Interpersonal Problems; and Making 
Judgments Under Uncertainty. Measures of only the first two of 
these domains were used in this study. Examples of indicators for 
these two domains are presented in Figure 2. 

The two sets of domains, derived from two distinct bodies of 
literature, reflect teacher behaviors used for two distinct 
purposes (related to student achievement/conduct and student 
reasoning/problem-solving) , Both sets of skills reflect important 
elements of teaching. 

Method 

This study was a single-group pretest-posttest design. 
Correlations between the two sets of teacher performance variables 
were computed before and after the training in teaching higher- 
order thinking, and the effectiveness of the training program was 
investigated by testing for differences in mean performance scores 
before and after the training. 

Subjects 

Nineteen secondary school teachers in a mid-sized Florida 
school district participated in the study. The participating 
teachers represented a variety of content fields, including 
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science, mathematics, home economics, language arts, and social 
studies. All participants were volunteers for the project. 

Instrum entation 

Concurrent with the development of the knowledge base for the 
essential teaching skills, researchers developed a classroom 
observation instrument for the measurement of the concepts and 
indicators of those domains. This instrument, the FPMS Summative 
Observation Instrument, is used by trained and certified observers 
to code teacher performance. The summative instrument: has been 
tested in reliability studies, resulting in reliability 
coefficients ranging from 0.91 to 0.98 (Teacher Evaluation Study, 
1983-1984) ; has been normed on a representative sample of Florida 
teachers; and has gained substantial empirical evidence for 
predictive validity in five subject-specific studies (Teacher 
Evaluation Study, 1985-1986) . 

The instrument used to measure higher order teaching 
performance is a combination of concepts and indicators from the 
THOT domains of (a) Developing and Maintaining Flexibility and 
Student Awareness, and (b) Generating and Validating 
Generalizations. As with the FPMS summative instrument, data are 
collected on this instri^ment by coding indicators of teacher 
performance. The estimated intercoder agreement on the THOT 
instrument is 0.82. 

Training Program 

Training in the teaching of higher-order thinking took place 
in the fall of 1989. Nineteen teachers in the areas of science, 
mathematics, home economics, language arts, and social studies 
participated. 

Over a period of 11 weeks, three 4-hour training sessions and 
seven 2-hour workshop sessions were conducted. The purpose of the 
first three sessions was to introduce, define, and provide both 
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examples and nonexamples of THOT classroom performance skills. The 
training sessions were conducted in a lecture/interaction format. 
During the training, the participating teachers were asked to 
select a topic that was already in the regular curriculum, on 
which a higher order thinking lesson would be developed. 

The seven workshop sessions were then spent developing and 
critiquing thinking-skill lesson plans. Lessons were developed by 
the teachers and critiqued by the research team and peer teachers 
in each subject area. Several potential topics in each subject 
area were presented during those sessions. Examples of topics 
chosen were (a) conditions that make ladybugs more active 
(science) ; (b) reasons time is an important resource (home 
economics) ; (c) determining how to measure quadrilaterals 
(mathematics) ; and (d) finding out what would happen if marijuana 
was legalized (social studies) . 

The six.'-h training session was devoted to editing each lesson 
and simulating presentations prior to practice teaching each 
lesson to a group of students. Practice-teaching sessions were 
videotaped and the videotapes were critiqued by the entire group 
during the eighth and ninth sessions. 

The final editing of each lesson was conducted by subject- 
area subgroups during the Venth session, after which each teacher 
taught his/her lesson to a group of students. Each lesson was 
videotaped, and the videotape was later observed by trained 
observers using the FPMS Summative Instrument and the THOT 
instrument. These observations produced the posttest classroom 
performance data. 

Rasult 

Pretest Observations 

For the pretest observations, each teacher was observed on 
two occasions by two different observers using the FPMS Summative 
Instrument and on two occasions by two observers using the THOT 



-5- 

S 



instrument. 

The FPMS observations were scored by (a) scaling the coding 
frequencies for each item according to criteria established in the 
instrument's norming, and (b) summing the item scale scores for 
each of the instrument's scales. The descriptive statistics for 
the FPMS observations are provided in Table 1. Comparing the 
obtained FPMS scores to Florida's norming distribution on the 
FPMS, the FPMS Total Performance scores provided a range of 64 
percentiles? the Effective performance scores provided a range of 
63 percentiles; and the Ineffective scores provided a range of 99 
percentiles. Although the mean scores for the sample of teachers 
participating in this study was somewhat lower than mean for the 
norming population, an adequate degree of score variability was 
obtained. 

The THOT observations were scored by (a) computing the square 
root of the number of times each item was coded as the item scale 
score, and (b) summing the item scale scores for each of the 
instrument scales. The descriptive statistics for this instrument 
are presented in Table 2. The indicators for Problem Formulation, 
Flexibility, and Student Awareness were coded more frequently than 
the indicators for the Development of Explanations and Validation 
of Generalizations. 

The zero-order correlations between scores on the two 
instruments are presented in Table 3. The correlation between the 
T^tal THOT scores and the Total FPMS scores was essentially zero 
(r = -0.01). The FPMS Effective Scores showed a small negative 
correlation with the THOT scores (r = -0.22), and the FPMS 
Ineffective scores showed a moderate positive correlation (r = 
0.39), although neither correlation was statistically 
significantly different from zero with this sample. 

Posttest Observations 

In parallel with the data collection strategy used for the 
pretraining observations, each teacher was observed on two 
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occasions by two different observers with each observation 
instrument following the training. 

The descriptive statistics for the FPMS observations are 
provided in Table 1. The stability of the measures of essential 
teacher performance are evident in these data. The mean FPMS Total 
Performance score was 71.85, compared with 70.24 on the pretest, a 
difference which is not statistically significantly different from 
zero (t = 0.90, df= 18). Similarly, the mean Effective Performance 
score was approximately 2.4 points higher cn the posttest, a 
difference which is not statistically significant (%_ = 1.43, 
df = 18) ; and the mean Ineffective Perlormance Score was 
approximately one point lower on the posttest, a difference which 
is also not significantly different from zero (1; = -1.16, df=18) . 

The descriptive statistics for the THOT posttest observations 
are provided in Table 2. In contrast to the FPMS Observations, 
mean performance on all measured concepts was significantly higher 
(P<.05) at posttest than at pretest. For the Total THOT score, the 
mean performance at posttest was approximately 0.4 points higher 
than mean performance at pretest = 5.68, df = 18, p < .05). 

As an aide to interpreting the magnitude of the changes in 
measured teacher performance from pretest to posttest, the raw 
change scores (posttest score minus pretest score) and the effect 
sizes (raw change score divided by the standard deviation of the 
pretest scores) are presented in Table 4. The mean Total 
performance score on the THOT instrument at posttest was 2.47 
standard deviations higher than the n^aan pretest THOT score. All 
of the subscale scores showed substantial increases, ranging in 
effect sizes from 1.11, for the measure of Student Awareness, to 
3.15, for the measure of Development of Explanations. 

A final presentation of pretest-posttest differences is 
provided in Figure 3. In this figure, the percentage of 
observations in which each indicator of the THOT instrument was 
coded is presented. Increases in the proportion of participating 
teachers' use of THOT behaviors is evidenced for all indicators, 
except Encourages Verbalization. This indicator showed a decrease 
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from 47% of the pretest observations, to 28% of the posttest 
observations. 

The zero-order correlations between scores on the two 
instruments, based on the posttest observations are presented in 
Table 5. The correlation between the Total THOT scores and the 
Total FPMS scores was zero. The FPMS Effective Scores showed a 
small positive correlation with the THOT scores (r = 0.08), and 
the FPMS Ineffective scores showed a small negative correlation (r 
» -0.15), As with the pretest correlations, none of these 
coefficients differs significantly from zero. 

Discussion 

The sporadic evidence of THOT indicators in the pretest 
observations mirrors the results of Peterson et al (1990), that 
these behaviors do not occur often in classrooms. The substantial 
increase in the occurrence of these behaviors following the 
training program, an effect noted in the previous research, is 
similarly noted in these data. This outcome supports the 
effectiveness of the training program in providing the 
participating teachers with the behaviors associated with student 
acquisition of higher-order thinking skills. Of course, whether or 
not the participating teachers will continue to use these THOT 
behaviors in their classroom teaching awaits further research. 

An implication of the results of this study, is that the low 
frequencies of THOT behaviors on the pretest suggest that training 
in the teaching of higher-order thinking is necessary for many 
teachers. Fortunately, the increase in THOT behaviors exhibited at 
the posttest suggests that such training in the teaching of 
higher-order thinking will be effective. The participating 
teachers in this study were willing and able to both learn and 
demonstrate teaching skills associated with higher-order thinking. 
The identification of the need to teach children how to think 
(e.g., Ec^M g^tinq Americans for the 21st Centurv . 1984) must be 
matched with efforts to educate teachers in the specific 
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operations of teaching thinking. Identifying a need without 
identifying mechanisms (both preservice and inservice) to meet the 
need is unlikely to effect any improvement in education. 

Finally, the stability of the measures of essential teaching 
skills in the pretest-posttest comparison supports the hypothesis 
that these two sets of teaching skills are relatively independent. 
The fact that the FPMS observation instrument did not register the 
improvement in the performance of the participating teachers 
indicates that the measurement of teacher performance in THOT 
behaviors requires different instrumentation than the current 
instruments for the measurement of essential teacher performance. 
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Table 1 

Distribution of Essential Teacher Perf ormance Scores as 
Measured bv the FPMS Summative Instrument . 



Pretest Posttest 



FPMS Scale M SD M SD 



Domain 3 

Lesson Organization 


22. 


71 


2.31 


25 


.39 


3. 


03 


Domain 3 

Lesson Development 


15. 


58 


1. 06 


15 


.72 


1. 


56 


Domain 4 

Subject Presentation 


13. 


13 


0.78 


13 


.84 


1. 


31 


Domain 5 
Communication 


10. 


74 


1. 51 


10 


.18 


1. 


6S 


Domain 2 

Management of Conduct 


6. 


58 


0.65 


6 


.40 


1. 


22 


Effective Performance 


34. 


49 


3.98 


36 


.88 


4 . 


70 


Ineffective Performance 


35. 


75 


2. 09 


34 


,97 


2. 


52 


Total Performance 


70. 


24 


3-87 


71 


.35 


5. 


96 


Note: N = 19. Scores are based 
obser/ations for each teacher. 


on 


the average 


Of 


two 
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Table 2 

BistfifcUtion of Higher Oi ..er Thinking Teach er Performance 
Scores as Measured bv the TH QT Instrument . 



Pretest Posttest 



THOT Scale M SD M SD 



Problem Fornulation 


0.31 


0.25 


0. 


89 


0. 


40 


Development of 
Explanations 


0. 10 


0. 13 


0. 


51 


0. 


36 


Validation of 
Generalizations 


0.03 


0.05 


0. 


15 


0. 


17 


Flexibility- 


0.26 


0.20 


0. 


67 


0. 


32 


Student Awareness 


0.31 


0.27 


0. 


61 


0. 


32 


Total THOT 


0. 18 


0. 15 


0. 


55 


0. 


22 


Note: N = 19. Scores 
observations for each 


are based on 
teacher. 


the average 


of 


two 







-12- 

er|c 1 5 



Table 3 

Correlations Between Mai or FPMS Scores and THOT scores at Pretest. 



FPMS Scores 



Effective Ineffective Total 



FPMS Ineffective -0.31 

FPMS Total 0.86* 0.22 

THOT Total -0.22 0.39 -0.01 

* E < -05 
Note : N = 19. 
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Table 4 

Pretest and Posttest Raw Piffenenoes and E ffect Sizes for FTMS and IHOT Scores . 



FFMS Score 


Raw 

Differetnoe 


Effect 
Size 


THOT Score 


Raw 
Difference 


Effect 
Size 


Ortganization 


2.68 


1.16 


Ptrtolem 
Formulation 


0.58 


2.32 


Development 


0.14 


0.13 


Dev. of 
E>q)lanaticaTs 


0.41 


3.15 


Sbjct Presentation 


0.71 


0.91 


Validaticxi of 
Generalizaticns 


0.12 


2.40 


Ocsnmmication 


-0.55 


-0.36 


Flexibility 


0.40 


2.00 


Mngrant of csonduct 


-0.18 


-0.28 


Student Awareness 


0.30 


1.11 


ttZ reccxve 


I . J9 


0.60 


Total THOT 


0.37 


2.47 


Ineffective 


-0.78 


-0.37 








Total 


1.61 


0.42 









Not9 ; N = 19. 
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Table 5 

Correlations Betveen Maior FPMS Scores and THOT Scores at 
Posttest . 



FPMS Scores 



Effective Ineffective Total 
0.29 

0.91* 0.66* 

0.08 -0.15 0.00 

* E < .05 
Note ; N = 19. 



FPMS Ineffective 
FPMS Total 
THOT Total 
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Figure 1 

Domains and Indicators of Essential Teacher Performance 



Domain 


Sample Indicators 


Planning 


Analysis of content, use of instructional 
materials, specification of activity 
format, matching learner needs with 
instructional element 


Lesson 

Organization 


Begins instruction promptly, orients students 
to classwork, conducts review 


Lesson 
Development 


Asks questions requiring analysis, 
amplifies responses, provides for practice 


Presentation of 
Subject Matter 


Treats concepts, applies principles, applies 
academic rules, develops criteria and 
evidence for value judgments 


Communication : 
Verbal and Nonverbal 


Emphasizes important points, expresses 
enthusiasm, challenges students 


Management of 
Student Conduct 

. 


Stops misconduct, maintains instructional 
momentum 
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Figure 2 

Domains and Indicators of Teaching Higher Order Thinking 



Domain 


Sample Indicators 


Developing and 
Maintaining 
Flexibility and 
Student Awareness 


Warns against premature evaluation, provides 
incubation time, encourages verbalization 


Generating and 

Validating 

Generalizations 


Identifies and classifies variables, 
formulates generalizations, tests hypotheses, 
validates generalizations 
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Figure 3 

THOT Indicator Coding Pattern on Pre and Post Observations 



TMOT Indicator 



Problem Foraulation 



I'resents Aackground 



Cites Variables 



Poses Problem 



Uses Analogies 



States General izat ion 



Classifies 



Determine Testability 



Uses hypothesis 



Falsifies hypothesis 



Models variable control 



Plans experiinent 



Pet of vbs. 

With Coding Histogram 



Development of Esipirical 
Explanations 



55 
93 

26 

67 

47 
93 

13 
26 



26 
79 

10 
22 

16 
28 

05 
10 

03 
09 

03 
10 

16 
69 



ii 



I 
I 
I 



THOT Indicator 



Pet of Obs. 

With Coding Htstogr£ 



Validation of Generalizations 

Guard against overgerwral iistion 



Analyze explwvtory power 



Relate conclusion to knowl. 



Evaluate alternative concliaton 



flexibility 

Warn against premature eval 



Provide exploratory materials 



Allow thinking time 



Student Awareness 



Thinks aloud 



Makes content explicit 



Encourages verbalization 



Uses small groups 



03 
26 

08 

U 

05 
21 

00 
14 



08 
65 

24 
31 

S3 
86 



40 
50 

29 
60 

47 
28 

10 
88 



I 

i 

i 



Key: Wi Pretest Observations 
H Post test CKTservattons 



ERIC 



no 



BEST COPY AVAIUBU 



